Object detection system

ABSTRACT

An object detection system for a vehicle includes a camera vision module and a Lidar module. The camera vision module includes an imaging device viewing to the exterior of the vehicle and operable to capture image data representative of a scene exterior and forward of the vehicle. The Lidar module includes a Lidar device that, with the Lidar module mounted at a front exterior portion of the vehicle, scans a region forward of the vehicle that overlaps with the field of view of the imaging device. Based at least in part on processing of captured image data by an image processor and based at least in part on processing of Lidar data captured by the Lidar device, 3-dimensional and timing information relative to the vehicle of objects present exterior of the vehicle is algorithmically constructed. At least one individual object of the multiple objects is prioritized.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/196,076, filed Jun. 29, 2016, now U.S. Pat. No. 10,295,667,which is a continuation of U.S. patent application Ser. No. 14/319,164,filed Jun. 30, 2014, now U.S. Pat. No. 9,383,445, which is acontinuation of U.S. patent application Ser. No. 13/242,991, filed Sep.23, 2011, now U.S. Pat. No. 8,767,186, which is a continuation of U.S.patent application Ser. No. 12/266,656, filed Nov. 7, 2008, now U.S.Pat. No. 8,027,029, which claims the benefit of U.S. provisionalapplication Ser. No. 60/986,016, filed Nov. 7, 2007, which is herebyincorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION

It is known to detect objects or other vehicles with camera visionsystems. Examples of such vision systems are described in U.S. Pat. Nos.5,796,094; 6,097,023; 6,320,176; 6,313,454; 6,559,435; 6,831,261;6,396,397; 5,877,897; 6,498,620; 5,670,935; 6,806,452; 6,946,978;7,123,168; 7,004,606; 7,005,974 and/or 5,550,677, which are herebyincorporated herein by reference in their entireties.

SUMMARY OF THE INVENTION

The present invention provides a road object detection and trackingsystems that is based on the fusion of camera vision and Lidartechnologies. The system and concept can provide driver assistance andsafety functions, such as adaptive cruise control, stop and go control,pedestrian detection, front and rear collision warning, lane departurewarning, side object detection, rear object detection, side and/or rearand/or front blind zone detection, and/or the like.

For the above mentioned driver assistance and safety applications, it isimportant to detect and identify objects on the road, and is desirableto measure object distance and relative speed to the driver's ownvehicle (the host or subject vehicle). The proposed technology has theadvantages over others to fulfill these requirements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the Lidar and camera-based vision system ofthe present invention;

FIGS. 2A-C are side elevations of a vehicle having a Lidar andcamera-based vision system incorporated therein in accordance with thepresent invention;

FIGS. 3A-D are schematics of different means for steering the Lidarlight source and detector line of sight in accordance with the presentinvention; and

FIGS. 4A and 4B are images of the field of view of the imaging sensor orcamera of the Lidar and camera based system of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings and the illustrative embodiments depictedtherein, a combined vision and Lidar system 10 of the present inventionprovides a combination of the core of Lidar and camera based visionmodules (see FIG. 1). The camera vision module 12 is comprised of a lens12 a, an imager 12 b, a vision ECU 12 c and vision algorithm. The visionmodule detects and identifies objects in the field of view. The Lidarmodule or system 14 in general is comprised of a light source 14 a, alight detector 14 b, lenses for source and detector, a scanning device14 c, a modulation signal generator, a signal amplifier, and a signalprocessing unit 14 d. Lidar system 14 measures object distance bycomparing the modulation phase or calculating a time of flight of thelight from the object (such as by utilizing processes such as thosedescribed in U.S. Pat. Nos. 6,825,455; 7,053,357; 7,408,627; 7,405,812;7,379,163; 7,379,100; 7,375,803; 7,352,454; 7,340,077; 7,321,111;7,310,431; 7,283,213; 7,212,663; 7,203,356; 7,176,438; 7,157,685;6,919,549; 6,906,793; 6,876,775; 6,710,770; 6,690,354; 6,678,039;6,674,895 and/or 6,587,186, which are hereby incorporated herein byreference in their entireties). Lateral and vertical resolution of ascene can be realized by mechanically scanning the light beam in araster fashion, such as in a manner similar to known or conventionalLidar systems, such as those described in the patents incorporated byreference above. In the illustrated embodiment, and as shown in FIG. 1,the vision system camera 12 b is used to provide lateral and verticalresolution of the scene. The mechanical scanner does not need to performraster scanning; instead, the Lidar line of sight and/or source light isguided by the vision module to point to the objects of interest andmeasure the distances between the objects and the host or subjectvehicle, as discussed below.

The Lidar light source can be any suitable light source, such as, forexample, a laser operating in an infrared wavelength (or near infrared)and eye-safe, a LED or LED array operating in the infrared or nearinfrared region and eye-safe, and/or one or more vehicle headlamps.Optionally, for example, the laser or LEDs 14 a can be mounted as a partof the camera-Lidar module that is mounted behind the rear view mirror16 inside the vehicle compartment, such as shown in FIG. 2A. Optionally,the laser or LEDs 14 a′ can be mounted separately from camera-Lidarmodule, such as shown in FIG. 2B. Optionally, and as shown in FIG. 2C,the system may utilize the vehicle headlamp 18 as the Lidar light source14 a″. In this case, high frequency pulse width modulation may beapplied to the headlamp. Optionally, LEDs as the Lidar source can bebuilt inside the headlamp assembly. Clearly, the Lidar source maycomprise other light sources and/or may be disposed elsewhere at thevehicle while remaining within the spirit and scope of the presentinvention.

When using laser as the Lidar light source, the laser may be collinearto the detector line of sight and steered by the same steeringmechanism. It can also be steered by a separate steering mechanism,which is synchronized with the detector line of sight steering. Theboundary of the scanned light should match vision camera's field of viewboundary. When using LEDs as the Lidar light source, the LED lightshould be collimated and then steered, or the LED(s) may be designed tospread the light to cover the whole field of view.

To provide a steering mechanism to the Lidar light source and detectorline of sight, one can use any suitable steering means, such as, forexample, via one of the following methods: (i) move the lens 14 e in Xand Y translational directions by mechanical means (FIG. 3A); (ii) movethe detector 14 b in X and Y translational directions by mechanicalmeans (FIG. 3B); (iii) reflect light by two scanning mirrors 15 a, 15 bthat rotate in orthogonal directions that equivalently provide X and Ymovements (FIG. 3C); and/or (iv) reflect light by one or two micromirror array (MEMS) devices 15 a′, 15 b′ that steer light in X and Ydirections (FIG. 3D).

Optionally, the steering mechanisms described in FIGS. 3A and 3B can berealized by a “speaker coil” type actuator that moves the lens ordetector, or a piezoelectric actuator that drives the lens or detectorin translation motions, or a servo/step motor to drive the lens ordetector in translation motions, or any other suitable means.Optionally, the steering mechanisms described in FIG. 3C can be realizedby a “galvanometer” type scanning mirror, or a piezoelectric actuator todrive the lens or detector in rotation motions, or a servo/step motor todrive the lens or detector in rotation motions or other suitable means.

The basic sequence of the operation may include the following. Thevision camera may acquire an image frame or frames at a certain rate(such as, for example, about 30 frames per second (fps) or thereabouts).The vision processing algorithm processes image(s) and identifiesobject(s) of interest in the field of view of the imager or camera. Asignal or data indicative of the X and Y coordinates of the objects iscommunicated to a Lidar steering unit. The Lidar unit is steered todirect the light toward a first object based on the given coordinatesand the unit may measure the distance to the first object, and the Lidarunit may be steered to direct the light toward a second object orobjects and may measure distance to the next object or objects, and soon, until all objects and distances are measured. The system maycalculate speed data for all objects if necessary or desired, and mayfeed the distance and speed data back to an object identificationalgorithm to improve object identification and prioritization. Thesesteps may be repeated during operation of the system.

The system provides 3 dimensional and 1 timing information of a roadobject(s) relative to the vehicle that hosts the system. The informationprovided may include the X and Y coordinates of an object in thecamera's field of view, the distance (Z) between the object(s) and thehost vehicle, and the time of possible collision based on the speed ofthe object and the host vehicle.

The vision module acquires, processes and identifies objects that are ofinterest to the applications, such as vehicles, pedestrians, roadsidebuildings, traffic lights, traffic signs, tail lights and head lights,and/or the like. The vision system may also identify lane markers, curbsand/or road dividers to help to determine the lane that the host vehicleis in and where the other vehicles or objects are relative to that lane.Identifying if another vehicle is in the host vehicle's lane is usefulfor adaptive cruise control systems (ACC) and/or stop and goapplications and/or collision warning applications. Algorithms that havealready been developed in Lane Departure Warning systems can serve thatpurpose.

Lidar measures the distance from an object to the host vehicle. Byadding this distance information to the 2-dimensional object positioninformation, the algorithm can construct full 3-dimensional positioninformation of an object or objects. The timing information given by thevideo frames, along with the host vehicle speed information (read fromvehicle bus), enable the calculation of the object speeds. The distanceand speed information can also feedback to the vision algorithm to helpits object identification. Besides the X and Y coordinates of theobjects that the camera vision module gives, the distance and speedinformation add two more dimensions to identify objects. In addition,lane mark information adds critical information for applications likeACC and collision warning. Based on the objects' 4-dimensionalinformation and lane mark information, a map of road objects can begenerated.

Priority scores can be assigned to all objects in the image. Thepriority scores represent the danger or importance of the objectsdetected in a scene. The objects with high priority scores have one, afew or all of the following characteristics:

-   -   vehicles that are close to the host vehicle;    -   vehicles that are closing to the host vehicle in relative speed        and collision will happen in pre-determined time;    -   vehicles that are in the same lane of the host vehicle;    -   vehicles that are moving to the same lane of the host vehicle (a        vehicle cut in the lane at a close distance in front of the host        vehicle); and/or    -   a pedestrian moving toward the center of the view.

In traditional Lidar systems that provide and process a full field ofview, the beam is scanned in raster fashion, or multiple detectors arescanned in sequence, in which case an equal amount of time is allocatedto all of the objects in the field of view. However, the system inaccordance with the present invention can assign different amounts of“staring” time to the different objects. A longer time may be assignedto the objects with higher priority scores. The coordinates of an objectidentified by the imager sensor guides the steering mechanism of theLidar. The Lidar sight can track and stay on an object while the objectmoves in the camera's field of view. As can be seen with reference toFIGS. 4A and 4B, the coordinates (X, Y) may be given by the image sensorto an identified vehicle 20 in the field of view. The Lidar sight, whichis represented by the circle in FIGS. 4A and 4B, is guided to point tothe vehicle.

By staring for a longer time at one object or a limited number ofobjects in the field of view and avoiding time consuming mechanicalsteering across the scene, the Lidar can provide faster distance readingof one object or a limited number of objects in the field of view. Thesystem provides the vehicle control module or modules with fasterdistance inputs, and thus allows quicker reaction time or speed andenhanced control to enhance avoidance of a collision with objects suchas a pedestrian or another vehicle. For example, in an ACC application,the vehicle in front of the host vehicle and in the same lane has thehighest priority score and will be stared by the Lidar for the mostamount time and the system will update the distance data to the vehiclecontrollers more frequently, in order to maintain the safe distance andavoid a collision with the other vehicle. In pedestrian detectionapplication, a pedestrian's distance and relative speed should bemonitored to determine if his/her trajectory potentially runs into thehost vehicle. The pedestrians who have more probability of running intothe host vehicle are assigned higher priority scores and then they get agreater amount of the Lidar time and more frequent distance updates. Incollision warning applications, the highest priority score may beassigned to a vehicle in the driver's or host vehicle's lane that isapproaching to a position and speed that will collide with the driver'sown vehicle within a pre-defined time. The system stares at the vehiclefor most of the time and updates the distance information to the vehiclecontroller more frequently to allow the vehicle controllers to takequicker reaction. In stop-and-go applications, the system may track thenearest vehicle in front of the subject or host or equipped vehicle andin the same lane, and may identify, track and react to cut-in vehiclefrom next lanes. Those nearest vehicles or cut-in vehicles are assignedwith most of the staring time and updates of distance measurement at thehighest frequency in order to provide the vehicle controller a quickerreaction time or speed.

While the Lidar sight tracks high priority score objects for more time,it is also important to monitor low score objects at a reasonablefrequency. Some objects may grow to higher priority ones and someobjects may change to reduced priority scores. Continuous and real timemonitoring and score assessment is desirable during operation of thesystem of the present invention.

Unlike the raster scanning mechanism in known Lidar systems (which haveto scan a full raster field of view for each frame), the Lidar system ofthe present invention has advantages of faster steering and objecttracking capability that are unique compared to competing technologies.By nature, the objects being tracked move slow in the horizontal andvertical directions in the camera and Lidar field of view. So themechanical scanning takes much less time to track objects or hop orswitch between the objects. In addition, the system does not have towaste scanning time in most of the “empty” space or spaces in the fieldof view that have no objects of interest. In other words, the system ofthe present invention can provide faster scene sampling rate or trackingspeed than the regular raster scan Lidar system and multiple detectorsystem. On the other hand, for the same or similar tracking speed, thepresent invention may require a relatively slower mechanical scanningspeed, which may lead to a lower cost and higher durability of thescanning mechanics. Another advantage of the present invention is thatthe system of the present invention may provide a more efficient andaccurate distance measurement, because it can stay at or process anobject of interest for a longer time than the raster scanning Lidar doeson the objects. Traditional raster scanning Lidar evenly spreads thebeam in the full field of view and therefore has a shorter exposure timefor each and every object in the field of view. Without wasting time onraster scanning, the system of the present invention can provide higherdistance sampling rate than regular Lidar and Radar.

The system of the present invention may provide more accurate distancemeasurement because the Lidar beam can stay on an object for longertime. The system allows more pulses to be collected and specialalgorithms or techniques to be applied to enhance the detectionsensitivity and confidence level. For example, the system may use agated accumulation and average (Boxcar technique) of many pulses oflight to increase the signal to noise ratio. Also, the system of thepresent invention may provide more accurate distance measurement becausethe object detection in the two dimensional image helps steer the Lidarbeam in a more accurate angle. In addition, the distance reading of anobject increases the confidence level of the two dimensional visionobject detection, which in turn provides feedback to the Lidar andprovides a more accurate distance measurement.

Another benefit of being able to “stare” longer on the object is that itprovides enhanced performance in adverse weather conditions, such asrain or fog. One known disadvantage of known Lidar systems is thedifficulty the system has in seeing long distances through rain or fog.The light is scattered or diffused by the water droplets in the rain andfog and causes the return light signal to be too weak to detect.However, by staying longer on objects, the system of the presentinvention can gather more light pulses and apply special methods andalgorithms to enhance the detection sensitivity and confidence level.For example, one can use gated accumulation and average (Boxcartechnique) of many pulses of light to increase signal to noise ratio.

The beam or line of sight steering needs to provide the field of viewthat is needed for matching the camera field of view and applicationneeds. A focused Lidar source light and detection spot can deliverbetter resolution and higher sensitivity over known systems.

Camera vision provides high lateral and vertical resolution and themature imaging processing technology allows sufficient or enhancedobject identification, lane mark detection, and/or the like in highspeed by utilizing mature color CMOS or CCD imagers. Lidar, on the otherhand, provides excellent longitudinal resolution. Both are maturetechnologies with relative lower costs as compared with other competingtechnologies, such as Radar or the like. By combining these twotechnologies together, the present invention provides advantages inperformance (such as high resolution in lateral, vertical andlongitudinal dimensions; high detection and tracking speed; and reliableobject identification and tracking), costs (such as due to the abilityto combine circuits to share components; combine processor and memory;and lower speed mechanical scanning mechanics), size (smaller packagesize; main Lidar components can be integrated with the camera; and thesystem may be placed behind rear view mirror) and integration withexisting vision-based systems, such as LDW, IHC, AFS, TSR and/or nightvision systems and/or the like.

Applications of such a vision-Lidar fusion system include (i) adaptivecruise control (ACC), (ii) Stop and Go; (iii) pedestrian detection; (iv)collision warning at both front and rear sides of the vehicle; (v)predictive brake assist; (vi) side object detection, (vii) side blindzone object detection; (viii) rear blind zone object detection; (ix)parking assist; and/or (x) lane change aid detection (a side objectdetection system may include both blind zone and lane change aidfunctions) and/or the like.

Therefore, the present invention provides enhanced processing techniquesover prior art technology, including scanning Lidar (which scans thewhole field of view and has a slower tracking speed and is not as goodat identifying objects in a complex scene and is not as good atdetection in fog or rain); Radar (which is expensive and has a lowhorizontal resolution (only a few lobes) and is not as good atidentifying objects in a complex scene); stereo vision (which requirestwo cameras and needs a wide baseline and rigid body, requires complexstereo image processing and has its distance accuracy limited by thecamera separation); 2-D vision (which requires complex image processingand is not as accurate in measuring distances and does not providereliable distance information for variable sizes of detected vehicles);range imager (an imager technology that can measure object distance bytime of flight of the light pulses emitted by the light emitter, inaddition to regular 2D image, and with the distance sensing being at theimager chip level, which is a complex semiconductor sensor fabricationand is a higher cost system).

Optionally, the imaging sensor of the present invention may comprise atwo-dimensional pixelated imaging array having a plurality ofphoto-sensing pixels arranged or disposed or established on asemiconductor substrate. For example, the imaging sensor may comprise acomplementary-metal-oxide-semiconductor (CMOS) or a CCD imaging sensoror device or the like, and may utilize aspects of the imaging sensorsdescribed in U.S. Pat. Nos. 5,796,094; 6,097,023; 6,320,176; 6,313,454;6,559,435; 6,831,261; 6,396,397; 5,877,897; 6,498,620; 5,670,935;5,760,962; 6,806,452; 6,946,978; 7,339,149; 7,123,168; 7,004,606;7,005,974 and/or 5,550,677, and/or PCT Application No. PCT/US07/75702,filed Aug. 10, 2007 and published as International Publication No. WO2008/024639, and/or U.S. patent application Ser. No. 11/239,980, filedSep. 30, 2005, now U.S. Pat. No. 7,881,496; Ser. No. 11/105,757, filedApr. 14, 2005, now U.S. Pat. No. 7,526,103; and/or Ser. No. 10/534,632,filed May 11, 2005, now U.S. Pat. No. 7,965,336, and/or InternationalPublication Nos. WO 2004/047421 and/or PCT WO 07/053404, which arehereby incorporated herein by reference in their entireties.

Optionally, the system may include any imaging sensor or sensors, andmay utilize aspects of various vision or imaging or detection systems,such as, for example, blind spot detection systems described in U.S.Pat. Nos. 7,038,577; 6,882,287; 6,198,409; 5,929,786 and/or 5,786,772,and/or U.S. patent application Ser. No. 11/239,980, filed Sep. 30, 2005,now U.S. Pat. No. 7,881,496; and/or Ser. No. 11/315,675, filed Dec. 22,2005, now U.S. Pat. No. 7,720,580, and/or U.S. provisional applications,Ser. No. 60/638,687, filed Dec. 23, 2004; Ser. No. 60/628,709, filedNov. 17, 2004; Ser. No. 60/614,644, filed Sep. 30, 2004; and/or Ser. No.60/618,686, filed Oct. 14, 2004, and/or road surface detection systems,such as of the types described in U.S. patent application Ser. No.11/948,086, filed Nov. 30, 2007 and published as U.S. Publication No. US2008-0129541, and U.S. provisional application Ser. No. 60/872,270,filed Dec. 1, 2006, and/or reverse or backup aid systems, such asrearwardly directed vehicle vision systems of the types described inU.S. Pat. Nos. 7,005,974; 5,550,677; 5,760,962; 5,670,935; 6,201,642;6,396,397; 6,498,620; 6,717,610 and/or 6,757,109, and/or of automaticheadlamp control systems of the types described in U.S. Pat. Nos.5,796,094 and/or 5,715,093, and/or U.S. patent application Ser. No.11/105,757, filed Apr. 14, 2005, now U.S. Pat. No. 7,526,103, and/orU.S. provisional application Ser. No. 60/607,963, filed Sep. 8, 2004,and/or rain sensors or rain sensing systems of the types described inU.S. Pat. Nos. 6,250,148 and 6,341,523, and/or of other imaging ordetecting systems, such as the types described in U.S. Pat. Nos.6,353,392 and 6,313,454, and/or U.S. patent application Ser. No.11/948,086, filed Nov. 30, 2007 and published as U.S. Publication No. US2008-0129541; Ser. No. 12/171,436, filed Jul. 11, 2008, now U.S. Pat.No. 7,914,187, and/or Ser. No. 12/190,698, filed Aug. 13, 2008, now U.S.Pat. No. 8,017,898, and/or U.S. provisional applications, Ser. No.60/872,270, filed Dec. 1, 2006; Ser. No. 60/949,352, filed Jul. 12,2007; Ser. No. 60/956,633, filed Aug. 17, 2007, and/or PCT ApplicationNo. PCT/US07/75702, filed Aug. 10, 2007 and published as InternationalPublication No. WO 2008/024639, and/or PCT Application No.PCT/US08/78700, filed Oct. 3, 2008 and published as InternationalPublication No. WO 2009/046268, and/or PCT Application No.PCT/US08/76022, filed Sep. 11, 2008 and published as InternationalPublication No. WO 2009/036176, with all of the above referenced U.S.patents, patent applications and provisional applications and PCTapplications being commonly assigned and being hereby incorporatedherein by reference in their entireties.

Optionally, the imaging sensor may be suitable for use in connectionwith other vehicle imaging systems, such as, for example, a blind spotdetection system, where a blind spot indicator may be operable toprovide an indication to the driver of the host vehicle that an objector other vehicle has been detected in the lane or area adjacent to theside of the host vehicle. In such a blind spot detector/indicatorsystem, the blind spot detection system may include an imaging sensor orsensors, or ultrasonic sensor or sensors, or sonar sensor or sensors orthe like. For example, the blind spot detection system may utilizeaspects of the blind spot detection and/or imaging and/or indicatingsystems described in U.S. Pat. Nos. 7,038,577; 6,882,287; 6,198,409;5,929,786 and/or 5,786,772, and/or U.S. patent application Ser. No.11/315,675, filed Dec. 22, 2005, now U.S. Pat. No. 7,720,580; Ser. No.11/239,980, filed Sep. 30, 2005, now U.S. Pat. No. 7,881,496 and/or Ser.No. 11/933,697, filed Nov. 1, 2007, now U.S. Pat. No. 7,777,611, and/orInternational Publication Nos. WO 2007/005942 and/or WO 2008/051910,and/or U.S. provisional applications, Ser. No. 60/618,686, filed Oct.14, 2004; Ser. No. 60/853,850, filed Oct. 24, 2006; Ser. No. 60/918,089,filed Mar. 15, 2007; Ser. No. 60/970,687, filed Sep. 7, 2007; and/orSer. No. 60/857,025, filed Nov. 6, 2006, and/or of the reverse or backupaid systems, such as the rearwardly directed vehicle vision systemsdescribed in U.S. Pat. Nos. 5,550,677; 5,760,962; 5,670,935; 6,201,642;6,396,397; 6,498,620; 6,717,610; 6,757,109 and/or 7,005,974, and/or ofthe rain sensors described in U.S. Pat. Nos. 6,250,148 and 6,341,523,and/or of other imaging systems, such as the types described in U.S.Pat. Nos. 7,123,168; 6,353,392 and 6,313,454, with all of the abovereferenced U.S. patents, patent applications and provisionalapplications and PCT applications being commonly assigned and beinghereby incorporated herein by reference in their entireties.

Typically, customer specifications may require blind spot sensors, suchas blind spot radar sensors and the like, to detect when they are blinddue to the buildup of dirt, ice or snow in front of the sensor. This isalso true for side object detections sensors which include side blindspot and lane change aid sensors. In practice, for many of thesesensors, meeting those requirements can present a challenge, since thebuildup cannot be actively detected, but rather has to be inferred fromthe lack of radar returns over a longer time. That works adequately indriving environments that have a lot of natural returns or objects forthe system to detect as the vehicle travels along the road (such asguardrails and/or other vehicles and/or the like), but on some roads(such as some elevated roads where a sensor may not detect any object tothe side of the host vehicle for minutes), such a system may fail. Ifthe host vehicle is driven along such roads (such as some elevated roadsin Sweden where a blind spot detecting sensor may not see anything forminutes), the absence of such a detection may trigger the blockagedetection sensor. It is thus envisioned that such a blind spot detectionsystem be combined with an input from a forward facing camera of thevehicle (such as a camera that is part of a headlamp control system orlane departure warning system or object detection system or the like ofthe vehicle). Thus, a blockage detection at the BSD sensor may bedetermined by the lack of detection of an object after that object isdetected by the forward facing camera. For example, if the front sensoror camera detects that the host vehicle is passing another vehicle orstructure, but the blind spot detecting sensor does not detect theobject a short period of time later (depending on the speed of thevehicle), the system can determine that the blind spot detecting sensorhas a blockage, whereby the system can detect such blockages muchquicker than previously possible.

Changes and modifications to the specifically described embodiments maybe carried out without departing from the principles of the presentinvention, which is intended to be limited only by the scope of theappended claims, as interpreted according to the principles of patentlaw.

The invention claimed is:
 1. An object detection system suitable for usein a vehicle, said object detection system comprising: a camera visionmodule configured for mounting at an in-cabin side of a windshield of avehicle equipped with said object detection system; wherein said cameravision module comprises an imaging device viewing through the windshieldto the exterior of the equipped vehicle when said camera vision moduleis mounted at the windshield of the equipped vehicle, the imaging devicehaving a field of view forward of the equipped vehicle and operable tocapture image data representative of a scene in a direction of forwardtravel of the equipped vehicle; the imaging device comprising atwo-dimensional pixelated imaging array having a plurality ofphoto-sensing pixels arranged on a semiconductor substrate; a Lidarmodule configured for mounting at a front exterior portion of theequipped vehicle; wherein said Lidar module comprises a Lidar devicethat, with said Lidar module mounted at the front exterior portion ofthe equipped vehicle, scans a region forward of the equipped vehicle,the Lidar device operable to capture Lidar data representative of thescene in the direction of forward travel of the equipped vehicle;wherein the region scanned by the Lidar device overlaps with the fieldof view of the imaging device; a control comprising an image processor;wherein, based at least in part on processing at the control of imagedata captured by the imaging device and based at least in part onprocessing at the control of Lidar data captured by the Lidar device,multiple objects present within the region scanned by the Lidar devicethat overlaps with the field of view of the imaging device are detected;wherein, based at least in part on processing at the control of imagedata captured by the imaging device and based at least in part onprocessing at the control of Lidar data captured by the Lidar device,3-dimensional and timing information relative to the equipped vehicle ofthe multiple objects present within the region scanned by the Lidardevice that overlaps with the field of view of the imaging device isalgorithmically constructed; wherein the 3-dimensional and timinginformation relative to the equipped vehicle of the multiple objectspresent within the region scanned by the Lidar device that overlaps withthe field of view of the imaging device comprises (i) X and Ycoordinates relative to the equipped vehicle of individual objects ofthe multiple objects present within the region scanned by the Lidardevice that overlaps with the field of view of the imaging device, (ii)distance (Z) between individual objects of the multiple objects and theequipped vehicle and (iii) time of possible collision between individualobjects of the multiple objects and the equipped vehicle; and wherein atleast one individual object of the multiple objects present within theregion scanned by the Lidar device that overlaps with the field of viewof the imaging device is prioritized based on at least one selected fromthe group consisting of (i) proximity to the equipped vehicle of the atleast one individual object present within the region scanned by theLidar device that overlaps with the field of view of the imaging deviceto the equipped vehicle, (ii) speed relative to the equipped vehicle ofthe at least one individual object present within the region scanned bythe Lidar device that overlaps with the field of view of the imagingdevice and (iii) location relative to the equipped vehicle of the atleast one individual object present within the region scanned by theLidar device that overlaps with the field of view of the imaging device.2. The object detection system of claim 1, wherein the prioritized atleast one individual object present within the region scanned by theLidar device that overlaps with the field of view of the imaging devicecomprises a pedestrian.
 3. The object detection system of claim 2,wherein, based at least in part on processing by the image processor ofcaptured image data, trajectory of the pedestrian relative to theequipped vehicle is monitored.
 4. The object detection system of claim2, wherein, based at least in part on processing by the image processorof captured image data, trajectory of the pedestrian relative to theequipped vehicle that potentially intersects with the equipped vehicleis determined.
 5. The object detection system of claim 4, wherein, basedat least in part on processing by the image processor of captured imagedata and upon determination that trajectory of the pedestrian relativeto the equipped vehicle potentially intersects with the equippedvehicle, capture by the Lidar device of Lidar data related to thepedestrian is enhanced.
 6. The object detection system of claim 1,wherein the at least one prioritized individual object present withinthe region scanned by the Lidar device that overlaps with the field ofview of the imaging device comprises another vehicle.
 7. The objectdetection system of claim 6, wherein, based at least in part onprocessing by the image processor of captured image data, the othervehicle is determined to be the nearest vehicle in front of the equippedvehicle.
 8. The object detection system of claim 6, wherein, when theequipped vehicle is travelling in a traffic lane on a road the equippedvehicle is travelling along and based at least in part on processing bythe image processor of captured image data, the other vehicle isdetermined to be travelling in the same traffic lane that the equippedvehicle is in.
 9. The object detection system of claim 6, wherein, whenthe equipped vehicle is travelling in a traffic lane on a road theequipped vehicle is travelling along and based at least in part onprocessing by the image processor of captured image data, the othervehicle is determined (i) to be the nearest vehicle in front of theequipped vehicle and (ii) to be travelling in a traffic lane that isnext to the traffic lane the equipped vehicle is travelling in.
 10. Theobject detection system of claim 9, wherein, based at least in part onprocessing by the image processor of captured image data, the othervehicle is determined to be travelling in a direction of travel that issame as the direction of travel of the equipped vehicle.
 11. The objectdetection system of claim 10, wherein, based at least in part onprocessing by the image processor of captured image data, the othervehicle travelling in the traffic lane next to the traffic lane theequipped vehicle is travelling in is tracked and identified to be acut-in vehicle if the other vehicle moves from the next traffic lane tothe traffic lane the equipped vehicle is travelling in.
 12. The objectdetection system of claim 1, wherein, responsive at least in part toimage processing of captured image data by the image processor, a lightbeam emitted by a light source of the equipped vehicle impinges theobject present forward of the equipped vehicle.
 13. The object detectionsystem of claim 1, wherein the Lidar device comprises a light source, adetector and a scanner.
 14. The object detection system of claim 13,wherein the scanner comprises at least one mirror.
 15. The objectdetection system of claim 13, wherein the scanner comprises a micromirror array.
 16. The object detection system of claim 13, wherein theLidar device determines distance to objects present forward of theequipped vehicle via a time of flight calculation.
 17. The objectdetection system of claim 1, wherein the at least one prioritizedindividual object present forward of the equipped vehicle comprisesanother vehicle, and wherein, at least in part responsive to imageprocessing by the image processor of captured image data, speed of theother vehicle present exterior of the equipped vehicle relative to theequipped vehicle is calculated.
 18. The object detection system of claim1, wherein the at least one prioritized individual object is prioritizedbased on potential danger of the individual object to the equippedvehicle.
 19. The object detection system of claim 18, wherein the atleast one prioritized individual object comprises a pedestrian who ispresent in the field of view of the imaging device.
 20. The objectdetection system of claim 19, wherein a light beam emitted from theequipped vehicle at least one of (a) tracks the pedestrian while thepedestrian moves in the field of view of the imaging device and (b)stays on the pedestrian while the pedestrian moves in the field of viewof the imaging device.
 21. The object detection system of claim 19,wherein, at least in part responsive to image processing by the imageprocessor of captured image data, the pedestrian present in the field ofview of the imaging device is assigned a high priority when thepedestrian is moving toward the center of the field of view of theimaging device.
 22. The object detection system of claim 1, wherein theat least one prioritized individual object present within the regionscanned by the Lidar device that overlaps with the field of view of theimaging device comprises another vehicle, and wherein the other vehicleis assigned a high priority when the other vehicle has one or morecharacteristic selected from the group consisting of (i) is closing withthe equipped vehicle and collision will happen in a pre-determined time,(ii) is in the same lane as that of the equipped vehicle and (iii) ismoving so as to cut into a traffic lane the equipped vehicle istravelling in at a close distance in front of the equipped vehicle. 23.The object detection system of claim 1, wherein the front exteriorportion at which said Lidar module is configured for mounting comprisesa front fender portion of the equipped vehicle.
 24. The object detectionsystem of claim 1, wherein the front exterior portion at which saidLidar module is configured for mounting comprises a front headlamp ofthe equipped vehicle.
 25. The object detection system of claim 1,wherein the at least one prioritized individual object of the multipleobjects present within the region scanned by the Lidar device thatoverlaps with the field of view of the imaging device is prioritizedbased on (i) proximity to the equipped vehicle of the individual objectpresent within the region scanned by the Lidar device that overlaps withthe field of view of the imaging device to the equipped vehicle, (ii)speed relative to the equipped vehicle of the individual object presentwithin the region scanned by the Lidar device that overlaps with thefield of view of the imaging device and (iii) location relative to theequipped vehicle of the individual object present within the regionscanned by the Lidar device that overlaps with the field of view of theimaging device.
 26. The object detection system of claim 25, wherein theat least one prioritized individual object comprises a pedestrian. 27.The object detection system of claim 25, wherein the at least oneprioritized individual object comprises another vehicle.
 28. The objectdetection system of claim 1, wherein, responsive at least in part toimage processing of captured image data by the image processor, atraffic lane in which the equipped vehicle is travelling is identified.29. The object detection system of claim 28, wherein, responsive atleast in part to image processing of captured image data by the imageprocessor, where other vehicles and other objects are relative to thetraffic lane in which the equipped vehicle is travelling along isdetermined.
 30. The object detection system of claim 28, wherein,responsive at least in part to image processing of captured image databy the image processor, lane markers of the traffic lane in which theequipped vehicle is travelling are detected.
 31. The object detectionsystem of claim 28, wherein, responsive at least in part to imageprocessing of captured image data by the image processor, a curb of aroad along which the equipped vehicle is travelling is detected.
 32. Theobject detection system of claim 28, wherein the imaging device capturesimage data for an adaptive cruise control system of the equippedvehicle.
 33. The object detection system of claim 28, wherein theimaging device captures image data for an intelligent headlightingsystem of the equipped vehicle.
 34. The object detection system of claim28, wherein the imaging device captures image data for a lane departurewarning system of the equipped vehicle.
 35. The object detection systemof claim 28, wherein the imaging device captures image data for anadaptive front lighting system of the equipped vehicle.
 36. The objectdetection system of claim 28, wherein the imaging device captures imagedata for a traffic sign recognition system of the equipped vehicle. 37.The object detection system of claim 28, wherein the imaging devicecaptures image data for a stop and go system of the equipped vehicle.38. The object detection system of claim 28, wherein the imaging devicecaptures image data for a pedestrian detection system of the equippedvehicle.
 39. An object detection system suitable for use in a vehicle,said object detection system comprising: a camera vision moduleconfigured for mounting at an in-cabin side of a windshield of a vehicleequipped with said object detection system; wherein said camera visionmodule comprises an imaging device viewing through the windshield to theexterior of the equipped vehicle when said camera vision module ismounted at the windshield of the equipped vehicle, the imaging devicehaving a field of view forward of the equipped vehicle and operable tocapture image data representative of a scene in a direction of forwardtravel of the equipped vehicle; the imaging device comprising atwo-dimensional pixelated CMOS imaging array having a plurality ofphoto-sensing pixels arranged on a semiconductor substrate; a Lidarmodule configured for mounting at a front exterior portion of theequipped vehicle; wherein said Lidar module comprises a Lidar devicethat, with said Lidar module mounted at the front exterior portion ofthe equipped vehicle, scans a region forward of the equipped vehicle,the Lidar device operable to capture Lidar data representative of thescene in the direction of forward travel of the equipped vehicle;wherein the region scanned by the Lidar device overlaps with the fieldof view of the imaging device; a control comprising an image processor;wherein, based at least in part on processing at the control of imagedata captured by the imaging device and based at least in part onprocessing at the control of Lidar data captured by the Lidar device,multiple objects present within the region scanned by the Lidar devicethat overlaps with the field of view of the imaging device are detected;wherein, based at least in part on processing at the control of imagedata captured by the imaging device and based at least in part onprocessing at the control of Lidar data captured by the Lidar device,3-dimensional and timing information relative to the equipped vehicle ofthe multiple objects present within the region scanned by the Lidardevice that overlaps with the field of view of the imaging device isalgorithmically constructed; wherein the multiple objects within theregion scanned by the Lidar device that overlaps with the field of viewof the imaging device comprise a pedestrian; wherein the 3-dimensionaland timing information relative to the equipped vehicle of thepedestrian present within the region scanned by the Lidar device thatoverlaps with the field of view of the imaging device comprises (i) Xand Y coordinates relative to the equipped vehicle of the pedestrianpresent within the region scanned by the Lidar device that overlaps withthe field of view of the imaging device, (ii) distance (Z) between thepedestrian and the equipped vehicle and (iii) time of possible collisionbetween the pedestrian and the equipped vehicle; and wherein thepedestrian present within the region scanned by the Lidar device thatoverlaps with the field of view of the imaging device is prioritizedbased on at least one selected from the group consisting of (i)proximity to the equipped vehicle of the pedestrian present within theregion scanned by the Lidar device that overlaps with the field of viewof the imaging device to the equipped vehicle, (ii) speed relative tothe equipped vehicle of the pedestrian present within the region scannedby the Lidar device that overlaps with the field of view of the imagingdevice and (iii) location relative to the equipped vehicle of thepedestrian present within the region scanned by the Lidar device thatoverlaps with the field of view of the imaging device.
 40. The objectdetection system of claim 39, wherein, based at least in part onprocessing by the image processor of captured image data, trajectory ofthe pedestrian relative to the equipped vehicle is monitored.
 41. Theobject detection system of claim 39, wherein, based at least in part onprocessing by the image processor of captured image data, trajectory ofthe pedestrian relative to the equipped vehicle that potentiallyintersects with the equipped vehicle is determined.
 42. The objectdetection system of claim 41, wherein, based at least in part onprocessing by the image processor of captured image data and upondetermination that trajectory of the pedestrian relative to the equippedvehicle potentially intersects with the equipped vehicle, capture by theLidar device of Lidar data related to the pedestrian is enhanced. 43.The object detection system of claim 39, wherein the Lidar devicecomprises a light source, a detector and a scanner, and wherein thescanner comprises a micro mirror array.
 44. The object detection systemof claim 43, wherein a light beam emitted from the equipped vehicle atleast one of (a) tracks the pedestrian while the pedestrian moves in thefield of view of the imaging device and (b) stays on the pedestrianwhile the pedestrian moves in the field of view of the imaging device.45. The object detection system of claim 43, wherein, at least in partresponsive to image processing by the image processor of captured imagedata, the pedestrian present in the field of view of the imaging deviceis assigned a high priority when the pedestrian is moving toward thecenter of the field of view of the imaging device.
 46. The objectdetection system of claim 43, wherein, responsive at least in part toimage processing of captured image data by the image processor, atraffic lane in which the equipped vehicle is travelling is identified,and wherein, responsive at least in part to image processing of capturedimage data by the image processor, lane markers of the traffic lane inwhich the equipped vehicle is travelling are detected.
 47. The objectdetection system of claim 46, wherein, responsive at least in part toimage processing of captured image data by the image processor, a curbof a road along which the equipped vehicle is travelling is detected.48. The object detection system of claim 46, wherein the imaging devicecaptures image data for a traffic sign recognition system of theequipped vehicle.
 49. The object detection system of claim 43, whereinthe imaging device captures image data for an adaptive cruise controlsystem of the equipped vehicle.
 50. The object detection system of claim43, wherein the imaging device captures image data for an adaptive frontlighting system of the equipped vehicle.
 51. The object detection systemof claim 43, wherein the imaging device captures image data for a stopand go system of the equipped vehicle.
 52. An object detection systemsuitable for use in a vehicle, said object detection system comprising:a camera vision module configured for mounting at an in-cabin side of awindshield of a vehicle equipped with said object detection system;wherein said camera vision module comprises an imaging device viewingthrough the windshield to the exterior of the equipped vehicle when saidcamera vision module is mounted at the windshield of the equippedvehicle, the imaging device having a field of view forward of theequipped vehicle and operable to capture image data representative of ascene in a direction of forward travel of the equipped vehicle; theimaging device comprising a two-dimensional pixelated CMOS imaging arrayhaving a plurality of photo-sensing pixels arranged on a semiconductorsubstrate; a Lidar module configured for mounting at a front exteriorportion of the equipped vehicle; wherein said Lidar module comprises aLidar device that, with said Lidar module mounted at the front exteriorportion of the equipped vehicle, scans a region forward of the equippedvehicle, the Lidar device operable to capture Lidar data representativeof the scene in the direction of forward travel of the equipped vehicle;wherein the region scanned by the Lidar device overlaps with the fieldof view of the imaging device; a control comprising an image processor;wherein, based at least in part on processing at the control of imagedata captured by the imaging device and based at least in part onprocessing at the control of Lidar data captured by the Lidar device,multiple objects present within the region scanned by the Lidar devicethat overlaps with the field of view of the imaging device are detected;wherein, based at least in part on processing at the control of imagedata captured by the imaging device and based at least in part onprocessing at the control of Lidar data captured by the Lidar device,3-dimensional and timing information relative to the equipped vehicle ofthe multiple objects present within the region scanned by the Lidardevice that overlaps with the field of view of the imaging device isalgorithmically constructed; wherein the multiple objects present withinthe region scanned by the Lidar device that overlaps with the field ofview of the imaging device comprise another vehicle; wherein the3-dimensional and timing information relative to the equipped vehicle ofthe other vehicle present within the region scanned by the Lidar devicethat overlaps with the field of view of the imaging device comprises (i)X and Y coordinates relative to the equipped vehicle of the othervehicle present within the region scanned by the Lidar device thatoverlaps with the field of view of the imaging device, (ii) distance (Z)between the other vehicle and the equipped vehicle and (iii) time ofpossible collision between the other vehicle and the equipped vehicle;and wherein the other vehicle present within the region scanned by theLidar device that overlaps with the field of view of the imaging deviceis prioritized based on at least one selected from the group consistingof (i) proximity to the equipped vehicle of the other vehicle presentwithin the region scanned by the Lidar device that overlaps with thefield of view of the imaging device to the equipped vehicle, (ii) speedrelative to the equipped vehicle of the other vehicle present within theregion scanned by the Lidar device that overlaps with the field of viewof the imaging device and (iii) location relative to the equippedvehicle of the other vehicle present within the region scanned by theLidar device that overlaps with the field of view of the imaging device.53. The object detection system of claim 52, wherein the other vehiclepresent within the region scanned by the Lidar device that overlaps withthe field of view of the imaging device is prioritized based onpotential danger of the other vehicle to the equipped vehicle.
 54. Theobject detection system of claim 52, wherein, based at least in part onprocessing by the image processor of captured image data, the othervehicle is determined to be the nearest vehicle in front of the equippedvehicle.
 55. The object detection system of claim 52, wherein, when theequipped vehicle is travelling in a traffic lane on a road the equippedvehicle is travelling along and based at least in part on processing bythe image processor of captured image data, the other vehicle isdetermined (i) to be the nearest vehicle in front of the equippedvehicle and (ii) to be travelling in a traffic lane that is next to thetraffic lane the equipped vehicle is travelling in.
 56. The objectdetection system of claim 52, wherein the Lidar device comprises a lightsource, a detector and a scanner, and wherein the scanner comprises amicro mirror array.
 57. The object detection system of claim 56,wherein, responsive at least in part to image processing of capturedimage data by the image processor, a traffic lane in which the equippedvehicle is travelling is identified, and wherein, responsive at least inpart to image processing of captured image data by the image processor,lane markers of the traffic lane in which the equipped vehicle istravelling are detected.
 58. The object detection system of claim 52,wherein, responsive at least in part to image processing of capturedimage data by the image processor, a curb of a road along which theequipped vehicle is travelling is detected.
 59. The object detectionsystem of claim 52, wherein the imaging device captures image data foran adaptive cruise control system of the equipped vehicle.
 60. Theobject detection system of claim 52, wherein the imaging device capturesimage data for an adaptive front lighting system of the equippedvehicle.
 61. The object detection system of claim 52, wherein theimaging device captures image data for a traffic sign recognition systemof the equipped vehicle.
 62. The object detection system of claim 52,wherein the imaging device captures image data for a stop and go systemof the equipped vehicle.
 63. The object detection system of claim 52,wherein, when the equipped vehicle is travelling in a traffic lane on aroad the equipped vehicle is travelling along and based at least in parton processing by the image processor of captured image data, the othervehicle is determined to be travelling in the same traffic lane that theequipped vehicle is in.
 64. The object detection system of claim 52,wherein, based at least in part on processing by the image processor ofcaptured image data, the other vehicle is determined to be travelling ina direction of travel that is same as the direction of travel of theequipped vehicle.
 65. The object detection system of claim 52, wherein,based at least in part on processing by the image processor of capturedimage data, the other vehicle travelling in the traffic lane next to thetraffic lane the equipped vehicle is travelling in is tracked andidentified to be a cut-in vehicle if the other vehicle moves from thenext traffic lane to the traffic lane the equipped vehicle is travellingin.